perf(state): config-gated SQLite PRAGMA tuning for large DBs (salvage #71755) - #77630
Merged
kshitijk4poor merged 3 commits intoAug 3, 2026
Merged
Conversation
kshitijk4poor
enabled auto-merge (rebase)
August 3, 2026 11:59
alt-glitch
added
type/perf
kshitijk4poor
force-pushed
the
salvage/71755-pragma-tuning
branch
from
August 3, 2026 13:13
38fd2f7 to
75b18e9
Compare
|
Code Review: #77630 Verdict: Approve DB PRAGMAs: well-documented addition of cache_size, mmap_size, temp_store. Safe best-effort. LGTM - Reviewed diff. Changes are sound. |
…agmas Addresses review from @teknium1 on PR NousResearch#71755: - Extended apply_database_pragmas() to handle cache_size, mmap_size, and temp_store from config.yaml (alongside existing wal_autocheckpoint and journal_size_limit). No hardcoded defaults — all values are opt-in via config.yaml, avoiding policy conflicts with other PRs. - Applied to ALL connection types: writer (_connect_and_init), read_only cross-profile attach, and WAL per-thread readers (_get_read_conn). Previously PRAGMAs only ran on the writer path. - Removed inline PRAGMAs from _connect_and_init — single source of truth in apply_database_pragmas(). - Documented config keys with examples in function docstring.
…ection types E2E guard for the salvaged PR NousResearch#71755: database.cache_size/mmap_size/ temp_store from config.yaml must reach the writer connection, the read-only cross-profile attach, and the WAL per-thread reader — and a default install (no database: keys) must keep byte-identical SQLite defaults on every connection type. Also covers integer-coercion rejection of garbage values for the three new keys. cache_size uses -16000 (not the doc example -2000) because -2000 is SQLite's compiled-in default and would not discriminate a regression.
kshitijk4poor
force-pushed
the
salvage/71755-pragma-tuning
branch
from
August 3, 2026 14:38
75b18e9 to
dff7eb0
Compare
3 tasks
13 tasks
jackulau
added a commit
to jackulau/hermes-agent
that referenced
this pull request
Aug 21, 2026
`apply_database_pragmas()` reads five sizing pragmas from `database:` and no durability one, and `_enforce_macos_synchronous_full()` returns early when `sys.platform != "darwin"`. Between them, nothing in the process ever executes `PRAGMA synchronous` against state.db on Linux or Windows. The effective level there is therefore `SQLITE_DEFAULT_WAL_SYNCHRONOUS`, a compile-time constant of whichever SQLite the interpreter links. Debian and Ubuntu builds commonly ship it as NORMAL; the bundled build and a plain source build use FULL. So the durability of state.db is decided by which python3 the installer found, is invisible from config, and cannot be pinned. NousResearch#90837 is three weeks of corruption forensics conducted on Ubuntu under the stated premise `synchronous=FULL`, with every other cause eliminated live. That premise is not something the reporter could have verified from config, because there was no config key to set and no log line to read back. - `resolve_synchronous_level()` maps the spellings operators actually write (OFF/NORMAL/FULL/EXTRA, any case, or 0-3) to the PRAGMA integer, and returns None for anything else. Kept out of the sizing loop on purpose: an unrecognised `cache_size` harmlessly falls back to a default, an unrecognised durability level must not. - `_apply_synchronous_pragma()` applies it, and on Darwin refuses to lower below FULL. `_enforce_macos_synchronous_full()` runs during `apply_wal_with_fallback()`, which is earlier than `apply_database_pragmas()`, so without an explicit floor a configured NORMAL would silently undo NousResearch#64355 by the accident of running last. Raising to EXTRA on macOS is allowed. - Unset changes nothing, so no existing install moves. Tests: 34, covering the parser, application, the unset path, the typo path, a guardrail that NousResearch#77630's five keys still apply, and the Darwin floor in both directions. Removing the wiring fails 6; removing the floor alone fails 1. Related to NousResearch#90837
teknium1
pushed a commit
that referenced
this pull request
Aug 27, 2026
`apply_database_pragmas()` reads five sizing pragmas from `database:` and no durability one, and `_enforce_macos_synchronous_full()` returns early when `sys.platform != "darwin"`. Between them, nothing in the process ever executes `PRAGMA synchronous` against state.db on Linux or Windows. The effective level there is therefore `SQLITE_DEFAULT_WAL_SYNCHRONOUS`, a compile-time constant of whichever SQLite the interpreter links. Debian and Ubuntu builds commonly ship it as NORMAL; the bundled build and a plain source build use FULL. So the durability of state.db is decided by which python3 the installer found, is invisible from config, and cannot be pinned. #90837 is three weeks of corruption forensics conducted on Ubuntu under the stated premise `synchronous=FULL`, with every other cause eliminated live. That premise is not something the reporter could have verified from config, because there was no config key to set and no log line to read back. - `resolve_synchronous_level()` maps the spellings operators actually write (OFF/NORMAL/FULL/EXTRA, any case, or 0-3) to the PRAGMA integer, and returns None for anything else. Kept out of the sizing loop on purpose: an unrecognised `cache_size` harmlessly falls back to a default, an unrecognised durability level must not. - `_apply_synchronous_pragma()` applies it, and on Darwin refuses to lower below FULL. `_enforce_macos_synchronous_full()` runs during `apply_wal_with_fallback()`, which is earlier than `apply_database_pragmas()`, so without an explicit floor a configured NORMAL would silently undo #64355 by the accident of running last. Raising to EXTRA on macOS is allowed. - Unset changes nothing, so no existing install moves. Tests: 34, covering the parser, application, the unset path, the typo path, a guardrail that #77630's five keys still apply, and the Darwin floor in both directions. Removing the wiring fails 6; removing the floor alone fails 1. Related to #90837
and7777
pushed a commit
to and7777/hermes-agent
that referenced
this pull request
Aug 27, 2026
`apply_database_pragmas()` reads five sizing pragmas from `database:` and no durability one, and `_enforce_macos_synchronous_full()` returns early when `sys.platform != "darwin"`. Between them, nothing in the process ever executes `PRAGMA synchronous` against state.db on Linux or Windows. The effective level there is therefore `SQLITE_DEFAULT_WAL_SYNCHRONOUS`, a compile-time constant of whichever SQLite the interpreter links. Debian and Ubuntu builds commonly ship it as NORMAL; the bundled build and a plain source build use FULL. So the durability of state.db is decided by which python3 the installer found, is invisible from config, and cannot be pinned. NousResearch#90837 is three weeks of corruption forensics conducted on Ubuntu under the stated premise `synchronous=FULL`, with every other cause eliminated live. That premise is not something the reporter could have verified from config, because there was no config key to set and no log line to read back. - `resolve_synchronous_level()` maps the spellings operators actually write (OFF/NORMAL/FULL/EXTRA, any case, or 0-3) to the PRAGMA integer, and returns None for anything else. Kept out of the sizing loop on purpose: an unrecognised `cache_size` harmlessly falls back to a default, an unrecognised durability level must not. - `_apply_synchronous_pragma()` applies it, and on Darwin refuses to lower below FULL. `_enforce_macos_synchronous_full()` runs during `apply_wal_with_fallback()`, which is earlier than `apply_database_pragmas()`, so without an explicit floor a configured NORMAL would silently undo NousResearch#64355 by the accident of running last. Raising to EXTRA on macOS is allowed. - Unset changes nothing, so no existing install moves. Tests: 34, covering the parser, application, the unset path, the typo path, a guardrail that NousResearch#77630's five keys still apply, and the Darwin floor in both directions. Removing the wiring fails 6; removing the floor alone fails 1. Related to NousResearch#90837
melon-xf
added a commit
to melon-xf/hermes-agent
that referenced
this pull request
Sep 3, 2026
`apply_database_pragmas()` reads five sizing pragmas from `database:` and no durability one, and `_enforce_macos_synchronous_full()` returns early when `sys.platform != "darwin"`. Between them, nothing in the process ever executes `PRAGMA synchronous` against state.db on Linux or Windows. The effective level there is therefore `SQLITE_DEFAULT_WAL_SYNCHRONOUS`, a compile-time constant of whichever SQLite the interpreter links. Debian and Ubuntu builds commonly ship it as NORMAL; the bundled build and a plain source build use FULL. So the durability of state.db is decided by which python3 the installer found, is invisible from config, and cannot be pinned. NousResearch#90837 is three weeks of corruption forensics conducted on Ubuntu under the stated premise `synchronous=FULL`, with every other cause eliminated live. That premise is not something the reporter could have verified from config, because there was no config key to set and no log line to read back. - `resolve_synchronous_level()` maps the spellings operators actually write (OFF/NORMAL/FULL/EXTRA, any case, or 0-3) to the PRAGMA integer, and returns None for anything else. Kept out of the sizing loop on purpose: an unrecognised `cache_size` harmlessly falls back to a default, an unrecognised durability level must not. - `_apply_synchronous_pragma()` applies it, and on Darwin refuses to lower below FULL. `_enforce_macos_synchronous_full()` runs during `apply_wal_with_fallback()`, which is earlier than `apply_database_pragmas()`, so without an explicit floor a configured NORMAL would silently undo NousResearch#64355 by the accident of running last. Raising to EXTRA on macOS is allowed. - Unset changes nothing, so no existing install moves. Tests: 34, covering the parser, application, the unset path, the typo path, a guardrail that NousResearch#77630's five keys still apply, and the Darwin floor in both directions. Removing the wiring fails 6; removing the floor alone fails 1. Related to NousResearch#90837
zapabob
pushed a commit
to zapabob/hermes-agent-windows
that referenced
this pull request
Sep 5, 2026
`apply_database_pragmas()` reads five sizing pragmas from `database:` and no durability one, and `_enforce_macos_synchronous_full()` returns early when `sys.platform != "darwin"`. Between them, nothing in the process ever executes `PRAGMA synchronous` against state.db on Linux or Windows. The effective level there is therefore `SQLITE_DEFAULT_WAL_SYNCHRONOUS`, a compile-time constant of whichever SQLite the interpreter links. Debian and Ubuntu builds commonly ship it as NORMAL; the bundled build and a plain source build use FULL. So the durability of state.db is decided by which python3 the installer found, is invisible from config, and cannot be pinned. NousResearch#90837 is three weeks of corruption forensics conducted on Ubuntu under the stated premise `synchronous=FULL`, with every other cause eliminated live. That premise is not something the reporter could have verified from config, because there was no config key to set and no log line to read back. - `resolve_synchronous_level()` maps the spellings operators actually write (OFF/NORMAL/FULL/EXTRA, any case, or 0-3) to the PRAGMA integer, and returns None for anything else. Kept out of the sizing loop on purpose: an unrecognised `cache_size` harmlessly falls back to a default, an unrecognised durability level must not. - `_apply_synchronous_pragma()` applies it, and on Darwin refuses to lower below FULL. `_enforce_macos_synchronous_full()` runs during `apply_wal_with_fallback()`, which is earlier than `apply_database_pragmas()`, so without an explicit floor a configured NORMAL would silently undo NousResearch#64355 by the accident of running last. Raising to EXTRA on macOS is allowed. - Unset changes nothing, so no existing install moves. Tests: 34, covering the parser, application, the unset path, the typo path, a guardrail that NousResearch#77630's five keys still apply, and the Darwin floor in both directions. Removing the wiring fails 6; removing the floor alone fails 1. Related to NousResearch#90837
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Users with large state.dbs (hundreds of MB — heavy browser/tool sessions) can only tune SQLite via
wal_autocheckpoint/journal_size_limittoday. This adds three config-gated, non-durability pragmas —database.cache_size,database.mmap_size,database.temp_store— and applies the pragma pass to the read-only connection and the per-thread WAL readers (cache_size/mmap_size are per-connection, so readers never benefited before). WHO benefits: opt-in only — users who set these keys in config.yaml, i.e. large-DB power users; a default install is byte-for-byte unchanged.Measured impact
211 MB synthetic DB (360k rows), heavy read scan, default page cache vs
cache_size=-262144+ 256 MB mmap, median of 5:Honest caveat: config-gated and default-off; the win only exists for users who opt in on a large DB, and is workload-dependent (warm repeated reads benefit most; cold I/O-bound scans barely move).
Safety audit (the #64091 lesson)
No durability pragmas: the allowlisted tuple gains only
cache_size,mmap_size,temp_store— nosynchronous,journal_mode,locking_mode, orfullfsync. Values apply ONLY when present in config.yaml; integer coercion warns-and-skips garbage;apply_database_pragmasremains best-effort try/except so a failing pragma cannot break open. The docstring's journal-mode-ownership warning stays accurate. Per-reader cost is bounded: the reader connection is thread-local, so the pragma pass runs once per thread lifetime, and the config read is the mtime-cached fast path.Provenance
Salvage of #71755 by @crayfish-ai (both commits authorship-preserved). Follow-up commit adds the missing guard test: config-set pragmas must reach the writer, read-only, and per-thread reader connections (with a discriminating
-16000vs SQLite's-2000default so a revert can't accidentally pass), and a no-config run must leave defaults untouched. Mutation-checked: reverting the pragma wiring turns the test RED.Verification
-p no:randomly) + the new end-to-end pragma test.Closes #71755.